OBSTRUCTIONS TO SHELL ABILITY 



Michelle L. Wachs 

Abstract. We consider a simplicial complex generaliztion of a result of Billera 
and Meyers that every nonshellable poset contains the smallest nonshellable poset 
as an induced subposet. We prove that every nonshellable 2-dimensional simplicial 
complex contains a nonshellable induced subcomplex with less than 8 vertices. We 
also establish CL-shellability of interval orders and as a consequence obtain a formula 
for the Betti numbers of any interval order. 



A recent result of Billera and Meyers [BM] implies that every nonshellable poset 
contains as an induced subposet the 4 element poset Q consisting of two disjoint 2 
element chains. (Throughout this paper shellability refers to the general notion of 
nonpure shellability introduced in [BW2].) Note that Q is the nonshellable poset 
with the fewest number of elements. Of course a shellable poset can also contain 
Q; eg., the lattice of subsets of a 3 element set. So the condition of not containing 
Q as an induced subposet is only sufficient for shellability; it does not characterize 
shellability. It is however a well-known characterization of a class of posets called 
interval orders and the question of whether all interval orders are shellable is what 
Billera and Meyers were considering in the first place. 

In this note we suggest a way to generalize the poset result to general simplicial 
complexes. We also give a simple proof of the poset result and prove the stronger re- 
sult that any poset that does not contain Q as an induced subposet is CL- shellable. 
This yields a recursive formula for the Betti numbers of the poset. 

We assume familiarity with the general theory of shellability [B W2] [BW3] . All 
notation and terminology used here is defined in [BW2 ]and [BW3]. 

The most simple minded conjecture one could make is that every nonshellable 
simplicial complex contains the induced subcomplex consisting of edges {a, b} and 
{c, d}, where a, 6, c, d are distinct vertices. A simple counterexample is given by the 
5 vertex simplicial complex consisting of facets {a, b, c}, {c, d, e}, {a, d}. Indeed the 
situation for simplicial complexes turns out to be much more complicated than it 
is for posets. 

The most natural thing to do next is to look for other "obstructions" to simplicial 
complex shellability. Is there a finite list? Below we see that the answer is no. Define 



Research supported in part by NSF grant DMS 9311805 at the University of Miami and by 
NSF grant DMS-9022140 at MSRI. 



1 



Typeset by Aj^S-T^i 



2 



MICHELLE L. WACHS 



an obstruction to shellability to be a nonshellable simplicial complex all of whose 
proper induced subcomplexes are shellable. The 4 and 5 vertex simplicial complexes 
given above are examples of one and two dimensional obstructions, respectively. 
The following observation was made by Stanley [S] . 

Proposition 1. For every positive integer d there is an obstruction to shellability 
of dimension d. 

Proof. Let K be the (d — 1) -skeleton of the simplex on vertex set [d + 3] = 
{1, 2, . . . , d+3} together with two (i-dimensional faces {1, 2, . . . , d+1} and {3, 4, . . . , d+ 
3}. We claim that K is a (i-dimensional obstruction. If K were shellable then by 
the Rearrangement Lemma of [BW2] there would be a shelling order in which the 
maximal dimensional facets come first; namely {1,2,..., d+1} and {3, 4, ... , d + 3} 
come first. But this is impossible because these two facets intersect in a face of 
dimension d — 2. Hence K is not shellable. 

Every proper induced subcomplex of K is either a simplex or consists of a single 
<i-face in a (d + l)-simplex together with the (d— 1) skeleton of the (d + l)-simplex. 
Certainly the simplex is shellable. Let J be the (d — l)-skeleton of the simplex on 
vertex set [d + 2] together with the face {l,2,...,d+l}. It is easy to see that 
lexicographical order on the facets of J is a shelling of J. (Lexicographical order 
on subsets of [d + 2] is defined by {a± < ■ ■ ■ < a^} < {b± < • • • < bj} if the word 
ai • • • cifc is less than the word b\ • • • bj in lexicographical order.) Therefore all proper 
induced subcomplexes of K are shellable. □ 

We now consider the following problem. 

Problem. Determine whether or not there is a finite number of d-dimensional 
obstructions to shellability for each d. If so, find bounds on the number of vertices 
that a d-dimensional obstruction can have. 

In this paper we solve this problem only for dimensions d = 1,2 and we leave 
open the problem for general d. 

Proposition 2. The only 1-dimensional obstruction to shellability is the complex 
J generated by facets {a, b}, {c, d} where a, 6, c, d are distinct. 

Proof. A 1-dimensional simplicial complex is shellable if and only if it has at most 
one connected component with more than 1 vertex. If K is a nonshellable 1- 
dimensional simplicial complex then let {a, b} be an edge in one component of 
K and let {c, d} be an edge in another component. The subcomplex induced by 
a, 6, c, d is J. Hence K is an obstruction if and only if K = J. □ 

Already in dimension 2 the situation is much more complicated. We use the 
following notation: For any subset U of V and simplicial complex K on vertex set 
V, let K(U) be the subcomplex of K induced by U. Also let the pure part of K, 
denoted pure(K), be the subcomplex of K generated by the facets of maximum 
dimension. For v G V, the link of v in K is denoted lk^(w) and is defined to be 
{F e K | FU{v} e K and v £ F}. For any v G V and subcomplex J of lk K (v), the 
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join of v and J is denoted v* J and is defined to be {F E K \ v E F and F\{v } E J}. 
The ith reduced simplicial homology of K over the ring of integers is denoted by 
Hi(K). 

Theorem 3. There are no 2- dimensional obstructions with more than 7 vertices. 

Proof. Let K be a 2-dimensional simplicial complex on vertex set V where \V\ > 7. 
Assume all induced proper subcomplexes are shellable. We shall show that K is 
shellable by showing that pure (if) is shellable and the 1-skeleton of K is connected 
(except for isolated points). That the 1-skeleton is connected follows immediately 
from the fact that no induced subcomplex consists only of a pair of disjoint edges. 
To prove that pure(if) is shellable, choose any vertex v of pure(if). Let 

K 1 =pme(K(V \{v})). 

Since the pure part of a shellable complex is shellable (by the Rearrangement 
Lemma of [BW2]), K\ is shellable. Let 

K 2 — v * pure(lkx(f))- 

We claim that pure(lkx(w)) is a connected 1-dimensional complex. If not there 
would be distinct vertices a,b,c,dE V \ {v} such that edges {a, b} and {c, d} are 
in different components of pure(lkx(f)). Since |V| > 5, the induced subcomplex 
K({v, a,b, c, d}) would be shellable which would imply that ^-K({v,a,b,c,d})( v ) is 
shellable since the link of any vertex in a shellable complex is shellable [BW3] . But 
this is impossible since ^K({v,a,b,c,d})( v ) has only two facets {a, b} and {c, <i}. It 
follows from this claim that K 2 is shellable and 2-dimensional. 

Next we dispose of the special cases that K\ is or 1-dimensional. Clearly K\ 
can't be O-dimensional since v belongs to a 2-face. If K\ is 1-dimensional then 
pure (if) = K 2 which is shellable. 

Now we can assume that K\ and K 2 are both shellable and 2-dimensional. Note 
that then 

pure(K) =K X UK 2 . 

Let 

A = K x n K 2 . 

We shall show that A is connected and 1-dimensional. Suppose it isn't. Then 
either (1) A = {0}, (2) A contains an isolated point or (3) there are edges in different 
components of A. For the first case, choose any a, , 6, c, d, e such that {a, b} E 
pure(lkft-(i;)) and {c, d, e} E K\. Since |V| > 6, K({v,a,b,c,d,e}) is shellable. It 
follows that {v, a, b} and {c, d, e} cannot be the only 2-faces of K({v, a, 6, c, d, e}). 
Hence there is a third 2-face F. Note v F because otherwise one of the other 
vertices of F would be in A. So F E K\. Since either a or b is a vertex of F as well 
as of pure(lk^(v)), a or b is a vertex of K\ n K 2 . Hence A cannot be {0}. 

For the second case, let a be the isolated point. Then K\ and K 2 contain 2- 
faces {a,c,d} and {v,a,6}, respectively, which intersect only at a. Since \V\ > 5, 
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K({v, a, 6, c, <i}) is shellable. This means that there is a third 2-face in the induced 
subcomplex that intersects each of the 2-faces along edges that contain a. If the 
third 2-face contains v then it is either {v, a, c} or {v, a, d}. This implies that either 
{a, c} or {a, <i} is in A, which contradicts the fact that {a} is a facet of A. Hence 
the third 2-face must be {a, 6, c} or {a, 6, d}. It follows that {a, 6} is a facet of A, 
which is still a contradiction. 

For the third case, suppose that {a, b} and {c, d} are edges in different compo- 
nents of A. Let x, y E V \ {v} be such that {a, 6, #} and {c, <i, are facets of K\. 
Since | V| > 7, J = F({v, a, 6, c, <i, y}) is shellable. Let 

B = (v * pure (lk j (v))) n pure(F({a, 6, c, <i, y})). 

Since B is a subcomplex of A, {a, 6} and {c, d} are in different components of 
B. It follows that Hq(B) ^ 0. Since i> * pure(lkj(i>)) is contractible, we also 
have Hi(v * pure(lkj(v))) = for all i. By the Rearrangement Lemma of [BW2], 
pure(F ({a, 6, c, d, x, y}) is shellable since the induced subcomplex K({a, 6, c, d, x, y}) 
is. Hence ^(pure(X({a, 6, c, d, x, y}))) = for i < 1. Since 

pure( J) = (v * pure(lkj(t>))) U pure(K({a, 6, c, (i, x, y})), 

by Mayer-Vietoris we have that i?i(pure(J)) = Hq(B) ^ 0. This contradicts 
the fact that J is shellable and 2-dimensional. Hence we may conclude that A is 
connected and 1-dimensional. 

Since A and pure(lk^(z;)) are connected we can get a shelling of pure(lk^(v)) 
by first listing the edges of A and then listing the remaining edges of pure(lkif (v)) 
so that each edge is connected to the previous ones. We claim that we can obtain a 
shelling of pure(-KT) by first listing the facets of K\ in the order given by the shelling 
of Ki and then listing the facets of K 2 = v * pure(Yk K (v)) in the order indicated by 
the shelling of pure(lkK(w)) in which the edges of A come first. Let Fi, F2, . . . , F n 
be the resulting ordered list of facets of pure(-KT). 

If Fi G K\ then it is clear that {F\ U F2 U • • • U F-i) n F is pure 1-dimensional. 
If Fi = {v,a,b} where {a, b} G A then it is easy to see that {a, b} is a facet of 
(A U F 2 U • • • U Fi_i) n F and that for all but the firsts inv*A, {v L a} or {v, b} 
is also a facet of (Fi U F 2 U • • • U F;_i) n Fj. Hence (F x U F 2 U • • • U F_i) n F, is 
pure 1-dimensional for all Fi E v * A. 

Now suppose Fj = {v, a, 6} where {a, b} A. Clearly either {v, a} or {v, b} is a 
facet of (FiUF 2 U- • •UFj_i)nFj; assume without loss of generality that {v, a} is the 
facet. If {b} is a facet of (Fi U F 2 U • ■ ■ U F_i) fl F, then 6 is in some 2-face {6, c, rf} 
of Fi. Consider the induced subcomplex L = K({v, a,b, c, d}). Since pure(L) 
is shellable, L contains one of the following 2-faces: {a, 6, c}, {a,b,d}, {v, b, c} or 
{v, 6, d}. The first two are impossible since {a, b} ^ A. The last two facets have to 
precede {v, a, 6} in the shelling because {6, c} and {6, <i} are in A. It follows that 
{v, b} G (Fi U F 2 U • • • U F_i) fl Fj which contradicts the assumption that {6} is 
a facet. Hence {&} is not a facet of (Fi U F 2 U • • • U F_i) fl Fj which implies that 
(Fi U F 2 U ■ ■ • U Fj_i) fl Fj is pure 1-dimensional. Therefore F 1? F 2 , . . . , F n is indeed 
a shelling of pure(F). □ 
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Corollary 4. Every nonshellable 2 -dimensional simplicial complex has a non- 
shellable induced subcomplex with n vertices where 4 < n < 7. □ 

Theorem 5. For each n = 5,6, 7, there is a 2- dimensional obstruction with n 
vertices. 

Proof. Let M n be the simplicial complex on vertex set {1, . . . , n} with facets {1, 2, 3}, 
{2, 3, 4}, . . . , {n — 2, n — 1, n}, {n — 1, n, 1}, {n, 1, 2}. For n > 5, M n triangulates 
a cylinder when n is even and a Mobius strip when n is odd. Hence M n is not 
shellable. We leave it to the reader to check that every induced proper subcomplex 
of M n is shellable when n < 7. □ 

The obstructions given in the proof of Theorem 5 are pseudomanifolds with 
boundary. We show next that a pseudomanifold without boundary cannot be an 
obstruction in any dimension. 

Lemma 6. Let K be a simplicial complex on vertex set V . Suppose that v G V is 
such that Kiy \ {v}) and lk^- (v) are shellable and no facet of lkje (f) is a facet of 
K(V \ {v}). Then K is shellable. 

Proof. First list the facets of K(V \ {v}) in any shelling order and then list the 
facets of v * lk K (v) in the order indicated by the shelling of lk K (v). It is easy to 
see that this is a shelling of K. □ 

Theorem 7. Let K be a simplicial complex for which every nonfacet face is con- 
tained in at least two facets. Then K is not an obstruction. Consequently, there 
are no obstructions that are pseudomanifolds (without boundary) or triangulations 
of manifolds (without boundary). 

Proof. The proof is by induction on dim.K. When dimK = 1 the result follows 
immediately from Proposition 2. Suppose dimK > 1 and every proper induced 
subcomplex of K is shellable. We will show that K must also be shellable. Let V 
be the vertex set of K and choose any v G V. Then K(V \ {v}) is shellable. It 
follows from the fact that every nonfacet face is in at least two facets of K that no 
facet of Dzk{v) is a facet of K(V \ {v}). 

To apply Lemma 6 we need only show that lk^ is shellable. Note that the 
property that every nonfacet face is contained in at least two facets, is inherited by 
lkft-(i>). Hence if lk^(v) is not shellable then by induction it contains an obstruction 
lkx (-{;)(£/"), where U C V \ {v}. We have that K(U U {v }) is shellable since it is a 
proper induced subcomplex of K. Since lk^(z;)(L r ) = 1^.k(uu{v})( v ) an d any link 
in a shellable complex is shellable, we have that \kK(v)(U) is also shellable, con- 
tradicting the fact that \]£k{v)(U) is an obstruction. Therefore Ikxiv) is shellable 
and by Lemma 6, K is shellable. So K is not an obstruction. □ 

A "pure" version of Lemma 6 is used implicitly in Provan and Billera's proof of 
shellability of matroid complexes [PB]. A matroid complex is a simplicial complex 
for which all induced subcomplexes are pure. Lemma 6 can, in fact, be used to 
prove the following stronger result. 
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Proposition 8. If every proper induced subcomplex of a simplicial complex K is 
pure then K is shellable. 

Proof. The proof is by induction on the size of the vertex set V. Suppose that K 
is not a simplex. Let F be any <i-dimensional facet of K where d = dim if. Choose 
v E V\F. Clearly K(V \ {v}) is pure <i-dimensional since it contains F. It follows 
that no facet of \kx(v) is a facet of K(V \ {v}). 

By induction, K(V \ {v}) is shellable. To apply Lemma 6, we need only show 
that lk K (v) is also shellable. For any U C V \ {v}, K(U U {v}) is pure. Since 
\k.K(v)(U) = ^-k(uu{v})( v ) an d any link in a pure complex is pure, we have that 
lk?f (?;)(£/) is pure. Hence every proper induced subcomplex of \kx(v) is pure. It 
follows by induction that lkfc(f) is shellable. □ 

Remark. Provan and Billera [PB] prove that matroid complexes are shellable by 
showing that they are vertex decomposible. The proof of Proposition 8 given here 
is a slight modification of the Provan-Billera proof and also yields the conclusion 
that K is vertex decomposible, but in the nonpure sense described in [BW 3]. 

Define an obstruction to purity to be a nonpure simplicial complex for which all 
proper induced subcomplexes are pure. Proposition 8 extends the Provan-Billera 
result from matroid complexes to obstructions to purity. It turns out that there 
are really very few obstructions to purity in each dimension. 

Proposition 9. For each d > 1, every d-dimensional obstruction to purity has 
exactly d + 2 vertices. Moreover there exists a d-dimensional obstruction to purity 
for each d. 

Proof. Let K be a <i-dimensional simplicial complex with vertex set V. Suppose 
\V\ > d + 2 and all proper induced subcomplexes of K are pure. We will show that 
K is also pure. 

Let v G V\F, where F is any <i-dimensional face of K. It follows that K(V\{v}) 
is pure <i-dimensional. To show that K is pure we need only show that lk^-(v) is 
pure (d — l)-dimensional. Let G be a face of \k K (v). Then since K(V \ {v}) is 
pure (i-dimensional, G is contained in some (i-dimensional facet H of K(V \ {v}). 
Since H U {v} ^ V, it follows that K(H U {v}) is pure (i-dimensional. Since 
G U {v} G K(H U {v }), it follows that G U {v} is contained in a facet of dimension 
d in K(H U {v}). This implies that G is contained in a (d — 1) -dimensional face of 
\kx(v), which means that Ik^iv) is pure and (d — l)-dimensional. 

An example of a (i-dimensional obstruction to purity is given by the (d — 1)- 
skeleton of the simplex on vertex set [d + 2] together with the (i-dimensional face 
[d+1]. □ 

Lemma 6 also yields a simple proof of the shellability of interval orders which 
we give below. Recall that a bounded poset is a poset that has a minimum element 
and a maximum element 1. If P is a bounded poset then P denotes the induced 
subposet P\ {0,1}. The length of bounded poset P is the length of the longest 
chain from to i. For any a < b in P, the open interval {x G P | a < x < b} is 
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denoted by (a, b) and the closed interval {x E P \ a < x < b} is denoted by [a, b]. 
The order complex of P is the simplicial complex of chains of P and is denoted by 
A(P). 

Proposition 10. (Billera and Meyers) Every interval order is shellable. 

Proof. Let P be an interval order. By the well-known characterization of interval 
orders, P does not contain Q (the poset with two disjoint 2 element chains) as an 
induced subposet. We may assume without loss of generality that P is bounded 
and that P has more than one atom. 

The fact that P does not contain Q enables us to choose an atom a such that 
each of the covers of a is greater than some other atom. Since lk A (p) (a) = A((a, 1)), 
this implies that no facet of \k A ^(a) is a facet of A(P \ {a}). Also the interval 

(a, 1) and the induced subposet P \ {a} both inherit the property of not containing 
Q as an induced subposet. Hence by induction they are shellable. We conclude 
that A(P) and hence P are shellable by Lemma 6. □ 

Remark. Bjorner [B] has independently used the same idea to prove more generally 
that all interval orders are vertex decomposable. 

Another proof that interval orders are shellable can be obtained using the tech- 
nique of lexicographical shellability, cf. [BW2] . 

Theorem 11. Every bounded interval order is CL-shellable. 

Proof. Let P be a bounded interval order. Partially order the atoms of P by 
letting a -< b if a has a cover that is not greater than b. Antisymmetry and 
transitivity follow readily from the forbidden induced subposet characterization of 
interval order. It is straight forward to verify that any linear extension of ^ is a 
recursive atom ordering of P by induction on \P\. Therefore P is CL-shellable. □ 

For any bounded poset P let Pi(P) be the ith reduced Betti number of A(P). It 
is easy to see that in the partial order on atoms given in the proof of Theorem 11 
there is a unique minimum atom. We shall refer to this atom as the smallest atom. 

Corollary 12. Let P be bounded interval order of length > 2, let A be its set of 
atoms and let ao be its smallest atom. Then for i > 0, 

A(P)= Yl ft-i(M). 

aeA\{a } 

Proof. By [BW2, Theorem 5.9], (3i{P) is the number of falling maximal chains of 
length i + 2 with respect to the CL-labeling induced by the recursive atom ordering 
given in Theorem 11. So we need to describe these falling chains. Each falling 
chain from to 1 of length i + 2 is of the form {0} U c where c is a falling chain 
of length i + 1 from a to 1 for some atom a. We need to determine which atoms a 
and falling chains c from a to 1 are such that {0} U c is falling. The proof of [BW1, 
Theorem 3.2] produces a CL-labeling from a recursive atom ordering (although it's 
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done in the pure case in [BW1], it easily carries over to general case, cf. [BW2]). A 
maximal chain has a descent on the subchain — > a — > b if and only if b is greater 
than an atom that precedes a in the recursive atom ordering. This happens for 
every maximal chain through a 7^ an and for no maximal chain through an- Hence 
the maximal chains of the form {a} U c, where a 7^ ao and c is a falling chain of 
[a, 1] , are the falling chains of P. □ 

The problem of studying obstructions could conceivably be made easier by con- 
sidering special classes of simplicial complexes that are closed under taking induced 
subcomplexes. A natural class, suggested by Bjorner [B], is the class of flag com- 
plexes. One might ask whether the pair of disjoint edges is the only obstruction for 
flag complexes. It turns out that this is not the case. The obstruction M7 given 
in the proof of Theorem 5 is a flag complex. However obstructions M5 and are 
not flag complexes. Also the obstructions given in the proof of Proposition 1 are 
not flag complexes. So there might still be something interesting to say about flag 
complex obstructions. 
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